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A beneficial suggestion is a constructive idea dnsiignel to result 
in an improvement or economy in the operation of a business or. antiviw, 
submitted in writing by an employee, for the purpose of having it cone 
sidered for an appropriate award. : \ ‘ 

From past experience, it has been determined that the qaaktity: of 
acceptable beneficial suggestions averages from 25 to 30 pereent. Sone 
believe that this is a reasonable level of success; others try to by-pass 
the problem and, by various means of publicity, increase the total num \ 
ber of suggestions submitted. While the percentage of acceptable rangers 
tions will remain about the same, the absolute number of eoceptable ones 
increases. ‘i 

A fundamental axiom of beneficial suggestions systers ie that the 
employee on the job is in the best position to recognize the nose for im 
provements. Since the average worker is not always qualified for this 
task, it would be desirable to give him some facts known to contribute 
to the development of an acceptable beneficial suggestion. \ 

Thus the aim or object of this paper is to identify, single out, 
evaluate and place in rank order those fundamental components that actu- 
ally contribute to an acceptable beneficial suggestion. \ ' 

Since the writer is a Naval Officer, it was decided to limit the 
scope of this study to three Naval Shore Establishments in the vicinity | 
of Lafayette, Indiana. The three plants selected wore the U. S. Naval | 
Ordnance Plant, Indianapolis, Indianas the U. 8. Naval Ordnance Plant, h 
Forest Park, Illinois; and the U. 5. Naval Ammnition Depot, Crane, : 
Indiana. These plants are alike in that they employ approximately 1600 
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civilians and engage in light manufacture. The latter diffors from the 
former two in that it has the added feature of large ammmition storage 
facilities. 

The problem, the determination of the factors contributing to the 
developuant of an acceptable beneficial suggestion, wes attacked by the 
internal consistency method. Past files of beneficial suggestions sub~ 
mitted between the years, 1946 and 1950, were divided inte two groups: 
"acceptable" and "not acceptable." These wore analysed in the light of 
twenty-four carefully selected factors. 

After the data had been gathered from the plants and the frequency 
of each factor determined, anaiysia wes carried out by the use offs 

Q Percentage 

be Devalues 

Cc. Correlation 

d. Multiple correlation 

By the use of percentages, those factors appearing less then 5 por~ 
cent of the time in winning suggestions were immediately discarded. D« 
values and sere order coefficients of correlation were used to place the 
remaining fectors in rank order. 

In the case of the Naval Ordnance Plent, Indianapolis, those fac- 
tors that contributed the most to the development of an acceptable mig- 
gestion were: 


le Is there & measurable monetary saving? ” 
2. Has an operation been eliminated or aude easier? *” 
i 


3e lave inepections been eliminated? 
le Has quality been improved? " 
5S. Has & movonent been eliminated or mule easier? 
* Frequency of selection as a comtributing factor in the plants studied. 
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& Has material been conserved? 

7» Has waste been reduced? 

&, Have operations been combined? 

9. Has a delay been eliminated? 

10. Has machine time been reduced? 

li. Have working conditions been inproved? 
12. Has morale been boosted? 

13. Have accident hazards been reduced? 
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Multiple correlation wes explayed to determine what selected com 
bination of factore would result in the largest correlation with the erl- 
terion. The three factora, selected from the previous list, that resulted 
in the largest multiple coefficient were: 


Selected Itens Wultiple 
. Coefficient 
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Saving, monetary «+ se see eee eee ene e ohh 
Saving plus work conditions 2.666 sseesee se 0S 


Saving plus work conditions plus fewer delays .. 095 


When the study of the beneficial suggestions at the Nava). Ordnance 
Plant, Forest Park, was completed, a slightly different group of factors 
essociated themselves with the winning suggeetions: 


ie Is there a measurable sonstary saving? . 
2+ Has a tool been sucgested? a 
3. Mae an operation been elizinated or meade easier? 

3 


he Has quality been improved? 
5S. Has a jig or fixture been suggested? 8 
6 Has a movement been eliminated or made eagicr? 3 
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7+ Has machine time been reduced? 

8 Have accident hagarde been reduced? 

9- Hes waste been reduced? 
20. Has materiel been conserved? 
il, Has a delay been eliminated or reduced? 
12. Has housekeeping been iaproved? 
13. Have working conditions been improved? 
lie Has morale been boosted? 
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& similar list resulted from the HAD Crane data after the original 
array of factors wae reduced and then ranked by Devalue and/or coefficient 
of correlation: 

le Is there a measurable monetary saving? 

2. Has a tool been suggcested? 

3- Has quelity been improved? 

le Has an operation been eliminated or made sasicr’ 

5S. Mave operations been combined? 

6. Has e movement been eliminated or mde casior? 

7+ Has waste been reduced? 

8. Have health hazards been reduced? 

9. Have accident hazards been reduced? 

10. Has a jig or fixture been suggested? 

ll. Has material been conserved? 

12. Has a delay been eliminated or reduced? 

13. las housekeeping been improved? 

ls. Has morale been boosted? 

15. Have working conditions been improved? $ 

16. Is Government property better protected? 1 
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In the case of the last two activities, the technique of multiple 
correlation was not pursued. Thu mumber of beneficial surrestions avail-~ 
able was not sufficient to result in intercorrelation coefficients reli- 
able enough te produce a meaningful mailtiple coefficient. 

Final anelysie of the resulte indicated thet in all cases the 
Single important factor in a beneficial suggestion was a measurable mone~ 
tary saving, This factor did not stand alone; it was always aasociated 
with another factor such as elimination of operations or employment of a 
new tool, jig, or fixture. 

Human relations factors brought up the end of the list. Now thia 
is to be expected for several reasons. First, the average working man 
has had little experience in solving problems of tiis type. Second, the 
factors pertaining to human relations, by definition, encompass workers 
@s @ group and not as individuals. The average suggestion submitted 
along these lines benefited a few rather than sany employees. 

However, when the Wherry-Doolittle technique wae applied to deter~ 
mine what combination of factors resulted in the highest wiltiple ccef- 
ficient of correlation, it was observed that wiile o soasurable saving 
was still first, some huwn relations factor wes secont. 

In conclusion it can be enid that it is possible to macure and 
place in rank order those factors that contribute te a winning beneficial 
suggestion. 
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AM AMALYSIS OF TNE FACTORS CONTRIBUTING TO AN ACCEPTABLE 
DENGFICIAL SUGGESTION 


TETRODUCTION 

A beneficial suggestion ie @ constructive idea designed to result 
in an iaprovement or economy in the operation of a business or activity, 
gubmitted in writing by an employee for the purpose of having it con- 
sidered for an appropriate award. 

The United Stetes Navy, like wany civilian firms, uses a benefi- 
cial suggestion program to save money aml improve employee morale, This 
ie not new to the Navy. As early as 1918, the first Navy Beneficial Supg- 
gestion Progras was authoriszed.* on ) August 1919, Franklin Delano 
Roosevelt, the Acting Secretary of the Navy, issued a circular letter to 
navel activities cutlining the rules for a Beneficial Sugeestion Program. 

Unfortumately this program did not provide amything like the de~ 
sired quantity of suggestions, This is borne out by the record, The 
files from 1919 to 1932 indicate that only 200 awarde were wade at the 
Department level for employee suggestions accepted during that thirteon- 
year period.” 

This plan continued until the outbreak of World War Il, It then 
became obvious that to better prosecute the war effort, every available 
bit of manpower and material aust be put to use. The Beneficial Sugges- 
tion Program offered a partial selution. It is sufficient to say here 
that the systea was streanlined and functioned satisfactorily during the 
war years, shortly after World War II, Emsoutive Order 9017 set forth — 
“ ete Adeivel fe Be Sikes, Chiat wy Office of Industrial Rela~ 

tions, address before the Bashington Regional Conference of the 


National Association of Suggestion Syetem, Washington, D. ~~ 
21 May 198. 
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the present regulations governing the awards to Federal employees for 
meritorious suggestions and for exceptional or meritorious service.” 

The United States Navy has com o long way since the act of Com 
grese on 1 July 1918. On 25 October 199, the Department of the Navy 
was awarded a plaque by the National Association of Suggestion Systens 
for being the government establishment to shew the greatest increare in 
participation by ite civilian emloyess in the Beneficial Suggestion 
Program during the year, 198.4 During the period specified, Navy en 
ployees submitted 38,673 suggestions as corpared with 28,029 the year 
before, Compare this to the 280 evs:pestions subsitted betwoen the years, 
4919 end 1938. On veviewing the mgnitode of the individual amarde, the 
picture is just as impressive. Of the highest, awards reported paid, the 
Mavy ranked thind with en individual award af ¢20,000.° 

However, these are past secouplishnents} one mst continuously 
look forward. Tt 4a wll to recall ithe words of Francis P. Mathews, who 
said: "In this day of shrinking appropriations, it is sore than ever 
iaportant that we obtain for the country the utnogt in defense for every 
Navy dollar expended." With this in mind, an effort will be made to de~ 
velop procedures and techniques to sake the Havy's program more effective. 
Tet ot @ hicuat 19hé (Pothie Law 690, 77th Congrows) Beotion Te 


¢ Seciiav ~ OR 223408 iaealar Jeniay Ge ted 10 Hovember 1919. 
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TRE PRORLEM 

When one compares the functioning of tho Navy Departwent's sug- 
gestion system with that of private industry, it in noted that the over 
all comparison is quite favorable (Table 1). While it is not the pur 
pose of this paper te account for my cinor differences appearing in 
Table 1, it is observed that both civilian companies ant the Navy have 
had @ proportional increase in the number of beneficial suggestions sub- 
mitted. 

In the secom item, pereent adopted, the Navy tends to lead civil-~ 
jan firms. However, this is offset to som extent by a slight decline 
in those adopted between the years, 1917 and 1919. 

The third item, average award, is slightly greater for the Navy; 
yet the rate of shrinkage from 19k7 to 1949 indicates thet they will 
soon be nearly equal. 

Although the data is not complete for the last two items, it ap- 
pears that the Navy system compares quite favorably with industry. As 
& point of explanation, it mst be remembered that civilian firm heve 
nothing comparable to a maxinum on-the-gpot award. This is a limitation 
izposed by the Naval Civilian Personnel Instructions Twenty-five, which 
sete forth the rules for operating a Beneficial Suggestion Program, The 
gum of $251.00 represents en average maximum of i pereent of the com- 
panies surveyed. 

In view of Table 1 and the observation that moneys spent for de~ 
fense must ve spent wisely, it would seem thet a technique ov method to 
assiat in increasing the voluse of acceptable beneficial suggestions 
would be highly desirable. However, this is a controversial subject. 
Some authors feel that 25 to 30 percent acceptable suggestions is adequate. 
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Table 1 
A Comparison of the Navy Department's Beneficial Suggestion Program 


with Recent Survey by the National Association of Suggestion Systems 


U. S. Navy’ 


NASS Survey® | 


Percent Adopted 

aronct ant 
Maximum on the Spot 
Award $251.00 
Maximum Award on 
Survey 


9 | $12,104 


6 charles M, Otis-Jemes L, MoVittie, Presentation of 1949 NASS Survey of 
Suggestion System Operation, Oct. 24, 1949. 
Navy Beneficial Suggestion Program, Annual Operating Statistics for 
1944 - 1948, 
Fiseal 1949, 
Not Available, 


For example, Seinwerth®® says: "If an exployes can win en award for one 
“vibe tong’ vnc un ihee enessinanneealendetp diane 
forming excellently." Presurably, Seimverth does nob mean that every 
employee can win an award, but of all the suggestione subaitied by an em 
Ployee, one out of every three or four will wine The fact that there is 
no univergal agreement on this point is demonstrated by efforte expended - 
' @t least interuittently ~ to increase the nusber of acceptable suggestions. 

Two standard methods for obtaining additional acceptable surges- 
tions are: (a) to put on a concerted advertising or publicity campaign, 
or (b) to point out specific problems that require a solutions The for~ 
mer has the disadvantage that it is e@ temporary stimulont. While the 
total volume of suggestions received anc the accompanying paper work is 
guxe to increase, the percentage of acceptable suggestions remains about 
the same, Even though the latter approach usually obtains better re- 
sults, it becomes increasingly difficult to locate worthwhile probloms. 
The writer's opinion, which hes bee borne out in conversation with mom 
bers of several beneficial suggestion committees, is that the heart of a 
suggestion is in locating the actual problemy the solution can be secom- 
plished by standard techniques. 

Thus, we return to the fundamentel concept of a beneficial sug~ 
gestion; the employee on the job is in the best position to recognige the 
need for improvements closest to him Unfortumtely this does not make 
the employee qualified for the task. It woulki appear that a worker could 
ve assisted immeasurably if he was presented with a Tow factors mow to 
contribute to an he ee beneficial — 
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Thus it om be said that the aim or chject of thie paper is to 
identify, single out, evaluate and place in rank order those fundamental 
components that actually contribute to an acceptable beneficial sugrestion. 
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The writer, being a Naval Officer, is primarily interested in the 
functioning ef the Naval Shore Zeteblishuent. Thus it was decided to 
Limit the scope of this study to three mvel activities in the vicinity 
of Lafayette, Indiana. The tires plants seleoted wore the U, 5. Noval 
Qrinance Plant, Indienapolis, Indianay the U. S. Naval Ordoance Plant, 
Yorest Parky Illinois; and the U. 5. Naval Amamition Depot, Crane, 
Indienae In the future, these three activities will be referred to by 
their short titles: NOPI, NCPIP, and NAD Crane respectively. 

In any testing procedure, it is necessary to set up @ suitable 
criterion. In som instances this might be quite difficult. Sowever, 
from the statement of the probles, it seems to go without saying that 
‘the criterion is to be the final dichotowy of any submitted beneficial 
suggestion, acceptance or rejection. 

This problem ie to be attacked by a technique known as the inter~ 
nal consistency method. This consists of measuring available beneficial 
suggestions and correlating the resulte with the criterion - winning. 
 ‘Thwe the first step is to compile a battery of tests and test then for 
reliability and validity. 

Because ali beneficial sugrestions at the above plants are judged 
in accordance with Naval Civilian Personnel Instructions Twenty-five 
(NCPI 25), thie publication offered a positive starting point. Paragraph 
lel of NCPI 25 lists the general types of suggestions desired. This 
list of suggestions has been reproduced in Table 2, To this table has 
been added four colums which refleet those item: considered sost appli- 
 eable by HOPI, NOP-PP, MAD Crane and MOF (U. S. Neval Gun Factory, 
“Washington, Ds C.) to their partécular activity. It should be noted here 
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Table 2 


Beneficial Suggestion Topics Considered Most Applicable by Several Naval Activities 


Methods of Operation 

Quality of Product 

Working Conditions 

Housekeeping One 
Improving 

Tools end Machinery . 

Protection of Gvt. Property 

Employee Morale en's 

Utilization of Manpower 


one 


New Tools anc Machinery 
Devising 
New Methods ....e. 


Cost: ss ads 
Waste .... + 
| Reducing Fire Hezards . 
Health Hazards 
Accident Hazards: 


Duplication .. 
Eliminating Unnecessary Work 
Breakage ... 
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that any omission does not man that this topic has been considered un- 
iepertant. This table represents topics within the scope of the average 
employee, after giving due consideration to the nature of the activity, 
type of work aecompliched and the abilities of the worker. 

On reviewing Table 2, two points are immediately obvious: 

1. Each organization has indicated ite preferences by selecting 

& group of subseets considered =ost applicable te ite 
situations 

2. While many of these items are quite spectfie, several are 

@ite broad and tend to overlap. 

In view of these preferences and the tendency to overlap, it 
geomed desirable to use a battery of teste with a breakdown so fine- 
grained that the tests would be fumlanental and applicable to any set 
of circumstances. Table 3, which contains 2 tests used to evaluate 


er’ ted. aul 
An operation ie all the work carried out at essentially one 
place, or in the vernacular, from tote~box to totebox. 

2. Has & movemmnt been eliminated or sade casier? 
4& movement has many connotations aml runs the gamt from 
(a) amount of body used, (b) foot pedals, (c) bimanuelness, 
(4) @ye-hand coordination, (¢) handling requirements, te 
(£) weights lifted or forces necessary to operate equipment. 
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it M. &. Mundel, Systematic Motion ani Time Study, Prentice-Hall, 197, 
PP. bh-92 
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Table 3 
Twenty-four Tests Used t Ivaluate Beneficial Suggestions 


Has an operation been eliminated or mide easier? 
Has & movemnt been elizinated or nade easier? 
Has a delay been ¢liminated or recuced? 

Have countings been ¢lindimited? 

Have inspections been eliminated? 

Have operations been combined? 

Have movements been conbined? 

Have delays been combined? 

Yas machine time been reduced? 

Is more work accomplished during the machine cycle? 
In group work, does one mm hold up the work? 
Have fire hazards been reduced? 

Have health hazards been reduced? 

Have eccident haserds been reduced? 


15e Yeve working conditions beon improved? 
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Has waste been reduced? 

Has material been conserved? 

Nas qielity been isproved? 

Has morale been boosted? 

Has housekeeping been improved? 

Ie Governzent property better protected? 
Is there a measurable nonetary saving? 
Has @ tool been suggested? 

Has @ jig or fixture been suggested? 


* Not included in study made at NCPI 
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The only delays to be considered are those which are umavoide 
eble or inherent in the job. An example would be operator 
idleness while waiting on e mchine to comlate ite cycle. 
Having countings been eliminated? 

Here couwrtings are to include either verification against a 
standard (inspection purposes) or the deteraination of the quan 
tity present (inventory). 

Have ingpoctions bocn eliminated? 

Besides the usual comotation of inspection, statistical methods 
are to be included. 

Have operations been combined? 

Operation means the same as in Humber 1. Coshining of operations 
might be facilitated by changing (a) work order, (b) using dif~ 
ferent eqaipment, or (¢) changing layout. 

Have movements been combined? 

Movenent is to mean the same as in Number 2. The combining of 
movements could be brought about ty (a) changing the order of work, 
(bo) changing the layout, or (c) changing the quantity handled at 
one time 

Have delays been_conbined? 

Delay means the same as in Nusber 3. 


Has machine time beer reduced? 
The maning of this ie obvious, 


Another wey of etating this would be to consider a change in the 
mamemnchine phase relationship. This would permit the worker te 
accomplish sore hand work during running tim. 


case 
‘paivamp od Meee gue oh wt a mb drat wo oo 
valow a8 atelqsoo of eabion 8 oc gubvtar aktiy | 
& dunteys senha ied seals a ne 
woe tantra yo ong e mntemte Lame att 
a 

aaotereq> To gatetdwed of toda at aa eise oft omnis aobireeg® 
“Lib gatw (cd) ytabwo deow (s) guiganid qd hedadtiest of sj is ; 
caamlenenadedattcyens ore 
Ye grntdmoo of? jd eilldbicitecs tale Getentaabale , 
“ghey Yo winw of yalgnedo (a) qi tuode Siyooud afb | oh 
Pans oy tn (0) ura tnt 6 ei 


bar 


myn 7 4a 


£ wodngll at ao-omss oft seam yal eye any 
ry 7 \ ! i 
‘yi : eS 5 4 - P ‘Ae i 


a a Pewee ae i” eh . ‘ sit a 
y : ; e ids ated Be 
edt ct agate 4 wattaene of od ino tte gattate taht 
OF wapecn elt Stoney Alvew Whee: saivened aire wand 
wld peta othub deve ead ewe i 


te. \, ae 
7 are | 
eh << 0) 
- - 7 4 7 
co (a 
~ ‘ ; ; Pa os | 
. i) at aa , 
. f } — Ll | 


In group work, does one man ap the work? 
then work performed during the cycle is wmeveniy distrilted, 
the wan with the most work to aecamplim: costrolis te ovecle 


This ie te include accidents to personnel anly. Accidents 
involving material, waste and Government property are in« 
cluded under other headings. 

liave working conditions been improved? 

Working conditions are the physical surroundings of the jobs 
Some evidence indicates that improved working conditions will 
improve morale." Thus, in gathering data, whenever this fac~ 
tor received a tally mark, Item 19 also recoived « tally sark. 
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Waste, as used hore, is synonymous with opoilage. It is the 
matorial rendered unmsable due to inadequacies of equipment, 
lack of sufficient omploywe skill and characteristics ine 
herent in the job. 

17. Has material been conserved? 

Material conservation ja the salvaging or reclaiming of 
materials having only scrap value, by a better use of pre- 


ory, Second Edition, Prentice~ilall, Ince, 
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peritly exployed uaterial and the omployaent of substitates, 
This item includes appearance, duproved Smetioning and longer 
life. 

Hes morale been improved? 

Morale is used in its collective sense here. It is the group 
state of wind or attitude due to the adoption of a particular 
suggestions It is not te be construed with the individual em 
ployeo's attitude due to subsitting a prise-winning suggestion. 
Has housekeeping been improved? 

This does not include any of the cleaning and preventive mine 
tenance detaile that are covered by maintenance personnel] on a 
predeternined schedule. 

is Govermunt property better protected? 

Besides the usual types of loss and dammge, also included are 
improved techniques for maintaining buildings and grounds, 
Is there a measurable nonetary savings? 

This item is self-explanatory; ite anewer, obviously, is a 
function of several variables. 

Has _a tool been suggested? 

Tool is used in a limited sense here. It includes any device 
that ie capable of working a miterial inte a desired shape and 
measuring devices; jigs ani fixtures are exciuded, 

Hes 8 jig or fixture ben suggested? 

& jig holds or is held ty the workpiece wiile guiding a cutting 
tool, Fixtures only hold or locate the workpiece. 
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Since the score on emy af the foregoing tests camnet be distri~ 
buted in the usual continuum but by only a simple dichotomy, "Yea" or 
"Yo", these tests will be referred to in the future as factors. 

While the actual gathering of the date was quite laborious, the 
light of all the supporting evidence. This usually consisted of the 
Original suzgestion, comments by the various departments on investiga- 
tion forms, miscellancous notes made by the recorder of the committee, 
and final disposition of the suggestion. As each factor appeared, a 
tally mark was recorded on a form much like Table 3. 

For purposes of this study, the final disposition of a sugges- 
tion £611 into one of two categories: 

1. Accepted 

& ‘Monetary award 

be Lotter of commndati an 
2. Rejected 

& Unsuitable 

be Technicality 

Since the purpose of tiis paper is to focus attention on the 
factors contritating to an acceptable sucgestion and not on the magnie 
tude of the award, suggestions thet received sonctary awards or letters 
of comendetion were grouped together. However, ail rejected sugces= 
tions were not considered te be in the sase category. For example, due 
te the instructions for operating a beneficial suggestion program, CPI 
2%, or any other set of rules for that wetter, a few good ideas are re~ 
jected on @ technicality alone. A typical situation would be whore an 
ides was in use more than sia monthe before being submitted, Thus, 
suggestions of this nature were not considered. 
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After the data had been gathered, it was analysed by several 
techniques, eae: of which had te be tempered with a certain amount of 
Judgment reflecting any wmugual cireumetances. Since it was desirable 
to reduce Table 3 to a shorter ani sore meaningful list as qiickly as 
possible, the following methods were enployeds 

1. Percentage 

2. Devalue 

3- Correlation 

ke Wualtiple correlation 

It was reslized at the outset that sone of the factors in the 
battery would appear infrequently in the final telly. Thue any factor 
that appeared less frequently than approximately 5 percent of the time 
in the acceptable breakdown was immediately deleted, This resulted in 
what will be referred to as an abridged list or table. 

Another, and sore elegant, method of detersdning the relation~ 
ship between the factors and the criterion is te determine the D-value 
and the coefficient of correlation, While Devalues and coefficients 
of correlation measure about the same thing, the ease of obtaining a 
Devalue is offset by the fact that the gere order coefficient is a com 
ponent that can be used in computing the coefficient of miltiple 
correlatione 

Yumerous statistical techniques for estimting the valicity of 
individual test item: have been reported in the literature. Long ani 
Sandiford!) have a technique wiose development they credit to Truman Le 
Kelley, The “Kelley technique" consists of the following steps: 

de Determine the percentage of succussful responses to a 

given item by members of the "high" eriterion group. 


John As “os axl Peter von gee “The Validation of Test Itema," 
Bulletin i ot sina alge 2 Pag Educational Research, Torontot 
eee ang of » 1935, Dre 16=50. 
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Determine the porcentace of successful. responser to ths 
item by weubere of the "Low" eriterfion group. 

Using the appropriate normal probability curve tables, find 
the etandarc gcore or « score value of the ordinate which 
subtends the pereentarce found in Step 1. 


Pir the corresponding value for the pereentage found in 
Step 2. 
Deteruine the discrimination value or validity index by 


subtracting the result cotained in Step 4 from the result 
obtained in Step 3. 


This systen involves a good deal of work 4£ there are sany test 
items, Lawshe has developed a ncmograph4, Figure 1, which reduces the 
time necessary to estimte Devalues or discrimination valuce which are 
suitably acourate. The nomograph is used by the following of the stepe 


below: 
Le 


Having determined the percentage af successful responses to 


& given ites by ueubers of the “accepted” criterion growpy 


locate this value on the senle marked: "Porcent of acnopted 
groupe” 

Raving determined the percentage of succeestul responses 

to the item by members of the “rejected” oriterion group, 
locate this value on the scale sarieds: “Percent of re~ 
jected group." | 

Lay @ streight-edge across the page eo that it will pase 


i Lewsey ores 


"A Monograph for ietimating the Validity of Test 
F. \ Paycholory y Vol. SOT, BG» by 
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A Nomograph for Estimating Devalues by C. H. Lawshe 


Fig. 1 
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A Nomograph for Estimating Devalues by C. H. Lawshe 


Fig. 1 


lie Read the D~value on the canter scale at the point where the 

Yor emaxple, let ue consider some of the NOPT data. Pagtor Xs, 
possessed by 72% of the accepted group and by 2u% of the rejected group, 
would be assigned a Devalue of 1.30. Another factor, Xy, possensed by 
_ «198 of the accepted group and by 11% of the rejected group, would have @ 

panies of Gals 

Lawahe also suggests that due to the asymptotic nature of the 
normal curve, the nomograph should not be used for values sualler than 
1 % nor greater than 99%. in any case, the particular D-value will be 
on the conservative side. 

Hot infrequently one oy both of the variables to be correlated 
are not contioucusly variable. Gne or both of the veriables may be 
qapable of ely being claseified into two or mre distinct classes.” 
Tor exauple, if wo wish to correlate the acceptance or rejection of a 
beneficial suggestion with the safety features therein, we are definitely 
limited to certain discrete categorise. Since the date in both vari~ 
ables is forced into two categories, or naturally belong in two, the ree 
mult is a fourfelé classification or tetracheric table. Let us take whet 
we might eall the responses to 391 beneficial suggestions to tee factors 
from the NOPI data, in which the response had to be "Yes" or "Mo." 

% Did the suggestion win an eward? 

igo Ie there & measurable sonetary saving? 

To use this data, the tetrechoric table is prepared as in Table h. 
However, using the date as it stands to comute the tetrachoric r is 
long and tedious. Guilford suygosts that Table & be converted to @ 


3, B. Guilford, 2 ode, MaGrewdiili Bouk Cou, Ince, 


First Uiition, 19%, 
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proportion bases as in Table § and the clagrams prepared wy Thuretene6 
andi hie associates be used to sclve for the tetrachoria r. Thur this 
was the technique selected to compute the coefficients of correlation 
to be used in the therry-Doolittie Test Selection Method. 

The task of determining an acceptable beneficial suggestion hy 
one of these factors alone is not as saticsfactory as the conlination of 
several factors into a battery. These factors, due te their very nature, 
measure one component of an entire suggestion. By referring to Table 6, 
page 25, it can be seen that cach factor measures success to a certain 
degree. At the same tim, theae very same factors have a low or nega- 
tive cormelation with each other. This would lemi one to assum thet 6 
selected cosbination of these factors would predict the chance of suce 
eevee better than one alone. 

The detemsination of this group of factors is quite a long pro~ 
cese by some asthods. However, the stamlard therry~Doclittle Test oe- 
lection Methed?? golves the problem with a minima of difficulty. This 
is a mthod for selecting a battery of tests, or factors in thin case, 
that will give the macimm shrunken miltiple correletion with the eri~ 
teriong that is, the maxciaum multiple correlation after a correction has 
been made for chance error added by each factor. These factors are se~ 
lected in the order of their contribution to the multiple. Ao a rule, 
the increase in the miltiple becomes lees axl less wile at the same 
time the chance error increases. A point is finally reached where owe 
ee ee, eee 
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tm page 7, reference was made to the reliability of « test. Te 
be reliable, a test must consistently measure some one aspest. In other 
words, repetitive use of a test on a particular subject aust yield os 
sentially the eaze results to have @ high reliabllity. This is best 
expressed in terms of the coefficient of correlation hetween the scores 
of the same or equivalent tests. Now, in the particular situation be- 
ing investigated, a unique comlition exists. This in because the "Yes" 
or "Ho" regponges to the applicability of the fectors in question coms 
directiy frou a beneficial mggestion that does or does not have the 
trait. Ine ones like this, nothing but o high reliability could re- 
@ult.e It is tiwrefore felt by the eriter that unless som caution is 
t© be cteerved, ons might construe a high wilidity, correlation with 
the eriterion, as indicating better results than are actually possible, 

Probably a better technique to be employed in evaluating the 
reliability would be in teres of two separate parties evaluating the 
gem beneficial wegeetion, After all, the problem is twofold ~ one of 
getting the same facte end one of 411 ths facts that are present. It 
mist be realised that in reading past files that are more than three or 
four years oli, no matter how carefully ali evidence is woighed, cor~ 
tain subtle pointe discussed by the action comzittes never become re- 
corded and are lost forever. 

This leads to the readily observable fnct that nearly 39 percent 
of the factors in Table 3, page 10, are so fine~graine! that they passed 
through the aleve of evalustion. Thies is not 6 serious difficulty; 
enough factors did come through to indicate specific results. Nowever, 
should any particular plant decide to mke a stuly such es thie, it ie 
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recomunied that the telly shoet, discussed on page 1), be completed uy 
the beneficial suggestion committee et the tia of agtual evalustion, 
For gli practical purposes, the discrimination value, used with 
@ reasonable amount of judgment, ic as satiefactory in this particular 
situation ag some cf the more elegant techniques af correlation. Thie 
is due primarily to the svall musber of suggestions studied. forever, 
Since this was the entire past record af NOPI and essentially 61) of 
NOP“¥P and NAD Crane, Little could be done to alter this situation. 
The effect of this small “" was particularly noticeable in the 
couputation of the tetrachorie coefficient of correlation where certain 
factors appeared relatively infreqently. Thurstone”! states: 


ere oe ee cee oe is sometines foun! thet two or 
three of the determinations can be made by using the central field 


of the diagren while me of the determinations must be onde by us- 
ing one of the cormors of the diagran. ceeltiniee Ie tok anaes 


tetrachoric eeekTisianh choca teh be Goanbal oh ali if tan éiei~ 
gion of either or both of the two distritutions is near the tall. 


Even with BH = 39] as in the case of HPT, the infrequent appear~ 
ance of a factor resulted in forcing the work dam into the corner of the 
computing diagrams. While it is realised tut sore laboricus mthaie of 
caleulation would result in 4 coefficient more accurate from the stand 
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point of arithmetic, unless one was certain with a high degree of confi- 
dence that the data were essentially accurate, amici expenditures of energy 
would be unwarranted. Thue it is felt that while the saguitade of the 
values of correlation and miltiple correlation might be open to com quee~ 
rect, However, to avoid the illusion of accuracy which the deta does not 
Justify, the use of miltiple correlation has been avoided for NOP-P? and 
WAD Crane. 
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Us Gs Ravel Ordnanse Plant, Indianapolis 

One of the first plants studied we the U. 5. Naval Ordnance 
Plant, Indianapolis, which exploys approximately 1,450 people. During 
the war years, HOPI was concerned primarily with the developmnt and 
- mamfecture of the Norden boubsight and airoraft lead-computing eights. 

As in the case of other military, as wall as civilian, establish- 
mente, the end of the war with Japan brought sasny changer in the opera~ 
tion of the Indianapolis Orinance Flant. Late in 1945, the station roe- 
verted to complete naval operation and was converted to apoomplish the 
mission originally intended, Lees, @ sioro establishsont for ronearch in 
and engineering developaunt of aviation ordnance fire control equipment; 
thon ond mulernisation of fire control instruments including redar et 

A study of 391 beneficial megeations wae sade at HOPI, of which 
130 wre considered acceptable ami 26] were rejected. These sucgestions 
covered an interval of time from January 1946 to February 1950. A 
graghic portrayal of the HOFT data and ite distribution are shown in 
Figure 2, page 50 in the Appendix. 

Table 3, on page 10, has been broken dow inte two tables. ‘the 
first, Table 6, includes ali the factors that appeared more frequently 
then 5 percent of the tim in the wiming suggestions. The first cclum, 
headed Renk, liste the factors in ander of both their Devalw ani corre 
lation with the criterion. The secon! colum, hemled Peetor Yunber, is 
the original factor order ae fowwl in Table 3. This colum ie retained 
throughout the discussion an & ready referumee t the factors in theiy 
Orisinal arbitrary position end alee as & numerical cdesiguation in ag 
following methemticnl computations. 
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4a one would expect, the single outetanding factor that exerts the 
moet influence on having a beneficial suggestion accepted is a masweble 
monetary euving. This factor never appears alons, but is aleers accoapenied 
by one or more of the remining factors. These factors divide theaselves 
inte thres general categories: 

i.e These pertaining to making 6 job eauior. 

2e Those that climinete waste and enable better utilisation 
of facilities. 

3» Those uesn relation factors with intangible scnetery 
benefits. 

Ac & complement t» Table 6, Table 7 lists the factors appearing in- 
frequently (less than 5 percent of the tim) among the accepted or winning 
miggestions, These factors reaiily divide themselves into two eaterories: 

il. Those previously sentioned Mine-grained factors that ere 
present but not included in the evaluation by the eup- 
gestion comalttes because other more gross factors better 
lend themselves to evaluation. 

2. Those factors that are suitable, but because of appli- 
cability and/or esployee's beckgrouml seldom receive con» 
sideration by an exployor, A sore complete ¢lacussion of 
this phase will be considered in the Conclusion, 

On contdiming the bwesiziowm of the factors, Table 6 has been sub 
divided into Teble 8 and Table 9. Table @ containg those fectors that 
can be associated vith tangible savings, wile Table 9 is restricted te 
those human relations factors that usually result in intangible benefits. 
The most striking aspect of these two tables ie that wile they heve 
been separated by tangible benefits reesived, they also fall inte this 
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very same grouping when ranked by Devalue and/or coefficient or correla 
tion, In other words, the three factors in Table 9 are the last tuwe 
factors in Table 6. 

As @ trie? aumary, it io wll to cheerve that @ seasursble scue<- 
tary saving is the bent single fector in a beneficial sugestion. On the 
other hand, considering only the facters that appear infrequently, those 
perteining to humm rolations ere ut the oxi of the list. 

As mentioned earlier, seldom do any of these factors appear alone. 
With tiie in aind, toree parallel studies wore aade of various cosbins- 
tions of these factora. Table 10 is made up of a sclected combination 
of factors teken fron Table 6, page 25. Necessarily, the flirwt factor 
here, measurable savings, is the save as in Table 6, Mowsver, if savings 
ere accompanied by a husm relations factor, improved working conlitions, 
the ecefficient of correlation increases from 04 to 9k. If a third 
faeter, vedwed delays, in added to these two, the coefficient of corre 
lation is increased slightly te .95. The determination of the best com 
bination of these factors depenis upon the intemcoorrelatian of these 
feetors with each other as well as with the oriterion. 

As @ parallel to Table 8 and Table 9, corresponding tables in- 
volving a combination of tangible and intanglole benefite have been pre~ 
pared respectively. Table ll, made up of only tengible benefite, has a 
measurable saving ag the first factor with a coefficient of correlation 
again of efke When the factor, reduced delays, hag boon added, the co~ 
efficient jJumpa to 230. By adding another facta, conservation af ma 
terials, the coofficient ig increased to 693. Table 12 picks up the three 
hunen relations factora, lovever, theme three « working conditions plus 
safety plus myale - oily raise the coefficient of correlation to .23. 
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Table 10 
Correlation Obtained between Evaluation on Selected Factors 
and the Criterion for 391 Beneficial Suggestions at NOPI 


at Selected Items ae SS 


Saving, measurable 


Saving plus work conditions 


Saving plus work conditions plus fewer delays 


Table 11 
Correlation Obtained between Evaluation on Selected Factors 
of the Abridged Battery of Tangible Saving Factors, Table 8 , 
and the Criterion for 391 Beneficial Suggestions at NOPI 


Paes Selected salad FR] somes me 


Saving, measurable 84 


Saving plus fewer delays 


Saving plus fewer delays plus save material 


Table 12 
Correlation Obtained between Evaluation on Selected Factors 


of the Abridged Battery of Human Relation Factors, Table 9 , 
and the Criterion for 391 Beneficial Suggestions at NOFI 


opehey Selected Items etl 


Working conditions 
Working conditions plus safety 


Working conditions plus safety plus morale 
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A veview of those tebles indicates thet a combination of tengible 
end intangible benefits reeulte in the highest miLtiple correlation with 
the eriterion. From a mathematics] viewpoint, this is due to a low or 
negative inter-correlation between factors. Practically, i+ scans the sug~ 
gestion has diversified qualities. 

The low multiple coefficient of correlation in Table 12 results 
from a relatively high inter-correlation between factors. fron a physi- 
eal stanipoint, thie means that each factor mamures about the same thing 
bat is designated by a different title. 


U. S. Naval Ordnance Plant, Forest Park 

The Naval Ordmance Plant, Forest Park, located in s western suburb 
ef Chicago, [llinoig, was originated by Congressional legislation at the 
eutect of the wartime emrgancy which, in effect, authorised the Secretary 
ef the Navy te provide additional ordnanoe aenufecturing facilities and 
equipment for the Kawy. This station was operated during llorld War Ti yy 
@ oubsidiary of tie American Can Company under a caxtract with the Ravy 
Departaent for the prodjuction of torpedoes, torpede parte, ax! assoublivg. 
Shortly after the cessation of hostilities, tie plant becam part of the 
permanent Neval Establishaent. Since conversion, the role of this eata~ 
thishaent has been primarily, in contrast to its wartinue production pace, 
the storage and modernization of torpedces anal tho mauufacture of now tom 
pedoes, spare parts and toolo, Likewise, the number of amploywes has been 
reduced from a wartime peak of approximately 6,500 to a present onboard 
count of approxiestely 1,300. 

A study of 322 beneficial suggestions subaittied between the widdie 
of 1947 and Pobruary 1950 wee wade at this plant. Of these 322 sugges 
tions, 142 were considered acceptable and 100 sere rejected. 4 graphic 
portrayal of the NOP-FP data and ite distribution are shown in Figure 3, 
page 51 in the Appendix. 

After these beneficial mgostions were evaluated, the results 
ware tabulated and put in a form similar to thowe of NOPI. Thus the face 
ters carteined in Table 3, page 10, were seperaied into two lists contain 
ing those that did and did net agpoer mare than S percent of the tim in 
winning beneficial euggestions. This resulted in Tables 13 and 1, respec- 
tively, Similarly, Table 13 wae further Groken down into those factors 
associated with tangible bemefite ami those with intangible ~ human rela 
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tions= benefite. This resulted in Tables 15 and 16 on pages 39 and 0 
respectively. 

Sines these tables ani the rank order of the factors therein paral- 
1e the study at MOPI, the reader is urged to rafer to pages 2% through 30 
far additional discussion and explanation of these tables. 
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U. &. Navel Awamition Depot, Crem 

The U. 5. Maval Ammunition Depot, Crane, comniesioned on 
1 Deventer 1942, is the lergost activity of ite kind in the Binth Naval 
District end the largest in utilised facilities in the United States. 
Located in the scutircontew. part of Indiana, 1¢ covers nearly 100 square 
wiles and employe approximately 2,100 civilians. 

Rewarkable though it is for ite cise, HAD Crane is exeng the new 
eet of ite type of activity. Ite remoteness from congested areas, ite 
ties, and ite hilly tepography meke it highly suitable for ongasine 

The U. S. aval Aummition Depet, Crane, has been assigned the 
following missions to manufacture, store, and overhaul anaunition, aw 
mmition components, and materialj end te supply thees items to the fleets 
ami other ordnance establishments, Although the huge storage facilities 
have accomacdated at one time thousands of poumle of high explosives, 
gnokeless powder, mdium-elsed bombe, and torpedces, tim menulecture and 
eeseubly of memnition is fully as inportent as the storage. 

In addition to the prodwition and storage functions of MAD Crene, 
it aleo has @ wiique supply fumetion. In addition to the tack of pro 
curing «materials for local use, thie department alec ia required to fa~ 
nigh “whelesale" lote of any of ite 129,000 axinancs spares te other ac» 
tivities and to the fleet. This extaile segregation, identifying, ant 
preserving these thousanis of parte. 

At the present time, NAD Grane is in « redwed operational aiatus 
with many of the ordnance plants laid wo for preservation ant others in 
& stemlby condition. 
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A study of 250 beneficial mugzestions, sabsitted botwen 
January 1947 ani February 1950, was aleo made at this activity. Of these 
250 suggestions, 126 were considered acceptable ani 122 were rejected. 
A graphic portrayal of tte HAD Crane data and ite distribution is shown 
in Figure , page 52 in the Appendix. 
ney as those af NCPI and NCP-IP, Thus Tables 17, if, 19 and 20 on pages 
h3, lib, 45, and L6 respectively group the factars by frequency of appear~ 
ange, infrequency of eppearance, monetary saving and nmin relations re- 
@pectively. As previously mentioned, the reader is urged te refer to 
pages 2 through 30 which cover the general aspects of those tables. 

While these tables are quite alile, it mst be cheerved that 
Table 20 on page b6 is ae twice es long a its shortest counterpart, 
Table 9 on page 29 for NOPIT, This is undoubtedly due to major emphasis 
pleced on safety ot NAD Cram, This, of course, is dus to the nature af 
the activity. While safety is being considered at ewory turn, it is 
bound to carry over to the more indirect phases euch as housekeeping and 
the protection of property. These facts coupled with Crane's relative 
isolation from a uajor Indians city contritate to this added interest 
ain the human relations elonents. 
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One of the met interesting features of this study hae been the 
general cimilerity of the end results. Most sigificant, end the factor 
ranked first by every setivity, is the masurable conetary saving - 
tangible benefit. All too frequently it is assumed that activities such 
@e thowe discussed ehove operate without due regard to oort. Yet, after 
evaluating 963 beneficial. su:gestions in three widely separated! planta, 
the single sost importent factor is saving money. 

As previously mentioned, this factor is never aloneg it is sup~ 
ported by awe or more of the cthors. While the supporting factor or 
factors cannot be singled cut by nam, 1% appears, in goneral, to be 
something associated with the employee's job and at a level he can cape 
with, Thia is bore cut by the factors contained in Tables &, 15, and 
19 on pages 25, 39, and LS pespectively. 

As would be expected, the previously mantioned fine-grained fac~ 
ters, appearing in Tables 7, 14, and 18 on peges 27, 37, and li respee~ 
tively were not caug’t in the mesh of tiie investigation. Possibly they 
were uot present. After all, such fine points as combining movementa, 
comhining delays, ate., are oxpected to be beyori the reasonable capa- 
bilities of the usugl operator end fall withia the sphere of management 
OF methods enginsering. 

Another group of factors in this category would be those par~ 
taining to humm relations, Tables 9, 16, and 20 am pages 29, 40, ant 
LG respectively. In broad term, factors such as morale and wording con~ 
ditions, which by definition encompass workers as a group rather than as 
individuals, are cleo beyond the usual scope of the operator on! fall 
within the domgin of mmageaent. 
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AS @ epecific oxmaple, ict us consider safety at WD Crane. “hile 
thie item receives preferential consideration, few of the suggestions were 
related to what one would consider the prevention of major disasters for 
an activity of that kinds the majority of Crane's suggestions were, in 
reality, the kind to be expected in any aanmulacturing establishaent. After 
& little thought, this is wiat one wuld expect. SAD Crane was devicned 
what possibility has the usual employee with only s farsing background of 
coming up with an acceptable euggostion of this type? if coxiitionn were 
prosont for him to accomplish this, he probably woiid not Live long enough 
to reach for a pencils 
| Thus it seems clear that an employos's best chance of eubultting 
@ Winning beneficial suggestion, yet containing & minimes mober of face 
tore, is tor 

de Improve his cw job, the one he is familiar with. 

Se Save the activity money in @ way that can be wensured 

with reasonable eave. 

3. Avoid the fine-grained factors that - while seasurable ~ 

only show sevings in mass production or contimous 

h. Avoid the humen relations factors + thay belong to 

nena gemante 
However, as mentioned on page 34, when several factors are ouployed, e 
— Coubinatdon ef beth tangible and intangible items reaulte in the highest 
correlation with the criterion « winnings 
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Have inspections been eliminated? 
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Has quality been improved? ea Weis a eee oe 


Has a movement been eliminated or made easier? 


Has material been conserved? ....... ; . 


Has waste been reduced? 
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Have operations been combined? ..... 


Have movements been combined? .... .« 
Has a delay been eliminated or reduced? 

Have countings been eliminated? Sg ve 
Have delays been combined? . - « « aa 
Is Government property better protected? 
Has housekeeping been improved? ane 
Has mechine time been reduced? ... 


Have working conditions been improved? 


Have health hazards been reduced? . . 
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Is more work accomplished during the machine 


Has an seerétion en eliminated or made canton’ 


So ar 
a2 


‘eycle? 


Has morele been boosted? AEA Cia ee gto: oe 


Have accident hazards been reduced? Dik ie es ete 


In group work, does one man hold up the work? | 
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Have fire hazards» been reduced? coee#eee ef 8 @ 


GMM Percentage of factor spplicable. 


[—} Percentage of factor not appliceble 


Fig. 2 


The percentage of suggestions possessing and not possessing vsrious factors at NOPI. 


142 Accepted 180 Rejected 
Suggestions Suggestions 


Is there a measurable monetary saving? . . , Ara 
Has a tool been suggested? e. 65 0 Oy ie ee he Woe os 
Has an operation been eliminated or made easier? ... 
Has quality been improved? ee eee eet er otis e tees 
Has machine time been reduced? .......-+-+ce.e 


Has a jig or fixture been suggested? ...... 2.2 
7. Has a movement been suggested or made easier? ..... 
8. Have accident hazards been reduced? .. ste eee 
7. men wenee GSE Tedueed? 2. ce ee te ee ee 
10. Has material been conserved? ..... se ec vecce 
ll. Has a delay been eliminated or reduced? ....... 
12. Has housekeeping been improved? ........s-«-e-cee 
13. Have countings been eliminated? .......2e+¢-+.-. : 
14. Have inspections been eliminated? ......2e2e-. 


15. Have movements been combined? aida” eaied a: is achak ada ve 


a6, Have Geluys teen coubined? . . 2566 se te ee ee 100% 100% 


O% ) 1% 
17. Is more work accomplished during the machine cycle? . . 100% eee 99% 


Of | O% 
18. In group work, does one men hold up the work? . «© - yo9g p———— >_> _}_F ____— jig 


19. Have fire hazards been reduced? ........-+..- 100% rp A Eig 
So Fae pyereidans Neat vebined? p< 6-2 ogg eo 
ai. "40 Government property better protected? .....-+ ogg oe AL — 97% 


22. Have working conditions been improved? .......e-. 95% 92% 
23. Have health hazards been reduced? ......+-+-+-+ 100 Ss 3% 97% 


24. Has morale been boosted? 6) gicjailel eee, yatetet ecistetee 6 ee ot 


GMS «Percentage of factor applicable | 


CJ Percentage of factor not applicable 


Fig. 3 The percentage of suggestions possessing and not possessing various factcrs st NOP-FP 
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ee. 128 Accepted 122 Rejected 

2 ae ’ Suggestions Suggestions 
1. Is there a measurable monetary saving? ....... 59% 7 SSS on 
2. Has a tool been su RORY Feo Sw ee eee ___ 30% £ 

. e gges 75h [ee ae ae eee 95% 
3. Has quality been improved? es 2 2 @- 2 ae e. RE: SS Tele 83% na RENESAS - 98% 

f (3% aaa 284 

4. Has an operation been eliminated or made easier? ... . 02 fe, | 

5. Have operations been combined? ............ 2 : 


92% 2. ay 51S aa 100% 
6. Has a movement been eliminated or made easier? ries : 


VAPOR eeeBOOn EOQNCGU? <. « « « s © 0 5 0 + + © « s\6 
8, Have health hazards been reduced? ......20-2e24-. 
9. Have accident hazards been reduced? .......4- 
10. -Has a jig or fixture been suggested? .....2-e ee 
ll, Has material been conserved? ......+.-.-+-s-ececee 
12,. Has a delay been eliminated or reduced? ....... 
13. Has housekeeping been improved? .......2e2e 
teehee wmoraie been boosted? ... ss. «+ «ss 6 sow é 
15. Have movements been Slonds. 545 cite ere 
16. Has machine time heen reduced? .....2-+-2e-eeee 
17, Have working conditions been improved? ....... 
18. Is more work accomplished during the machine cycle? . 
-19. Is Government property better protected? ...... 
20g shave délaye been combined? . . «se « © « « 0 «© 
21. Have inspections been eliminated? ......2.2+.2-6 
22. In group work, does one man hold up the work? ...-. 
23. Have fire hazards been reduced? .......-.2-+-2-6 
24. Have countings been eliminated? ......2+-e+ee8 
GMs Percentage of factor applicable 


CJ) Percentage of factor not applicable 
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Fig. 4 The percentage of suggestions possessing and not possessing various factors at NAD Crane 
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